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Purpose  of  Our  Research 


Can  we  improve  the  probability  of  a  program’s  success  through  a 
method,  to  be  used  by  PMOs,  that  produces  mutually  constrained  and 
aligned  program  acquisition  strategy  and  software  architecture? 

Why  this  is  important 

•  Software  is  increasingly  important  to  the  success  of  government  programs. 

•  There  continues  to  be  little  consideration  of  the  software  architecture  in  the 
development  of  either  the  system  architecture  or  the  program’s  acquisition 
strategy. 

•  Software  architecture  is  often  over  constrained  by  decisions  made  early  in  the 
acquisition  lifecycle  when  key  program  choices  are  being  made  -  negatively 
affecting  program  success. 


Alignment  among  the  software  and  system  architecture 
and  acquisition  strategy  does  not  occur  naturally 
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Interplay  of  Acquisition  and  Architecture 


& 

_ _ K 

?> 

monolithic  legacy  new  modular  architecture  with 

architecture  new  ar|d  legacy  capabilities 


Program  Manager 


Should  I  have  1  contractor,  or  2  or  3  or  6? 

If  1  contractor,  how  do  I  enforce  a  modular  architecture? 

If  multiple  contractors,  how  do  I  ensure  the  parts  fit  together? 
Can  I  migrate  legacy  to  give  me  a  quick  delivery? 
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Phase  1  Research:  Characterize  Failure  Patterns 


Reoccurring  patterns  of  failure 

•  Undocumented  Business  Goals 

•  Poor  Consideration  of  Software 

•  Unresolved  Conflicting  Goals 

•  Failure  to  Adapt 

•  Turbulent  Acquisition  Environment 

•  Overlooking  Quality  Attributes 

•  Inappropriate  Acquisition  Strategies 


Entities  and  relations:  the  way  it  should  be 


satisfies 


Phase  1  results  published  in  SEI  TN  CMU/SEI-2013-TN-014: 
“Isolating  Patterns  of  Failure  in  Department  of  Defense  Acquisition" 
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Phase  2  Research:  Explore  Acquisition  Quality 
Attributes 


Focus  research  to  start  filling  the 


Captured  75  scenarios  across  23 
programs 

>  Identify  candidate  acquisition 
quality  attributes  (AQA) 

>  Determine  how  to  express 
program-specific  AQAs 

>  Construct  AQA  scenarios 

>  Analyze  the  scenarios 

>  Build  a  prototype  workshop  to  elicit 
AQA  scenarios 


Phase  2  results  to  be  published  in  SEI  TN  CMU/SEI-2013-TN-026: 
“Results  in  Relating  Quality  Attributes  to  Acquisition  Strategies" 
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Candidate  Acquisition  Quality  Attributes  (AQAs) 


Original  candidates 


Acceptability 

Competitiveness 

Modifiability 

Accountability 

Contract  manageability 

Promptness  in 

Affordability 

Credibility 

reporting  problems 

Appropriateness  of 

Effectiveness 

Responsibility 

contract 

Evolvability 

Responsiveness 

Appropriateness  of 

Fairness 

Sensibility 

technology 

Flexibility 

Staffability 

Achievability 

Implementability 

Suitability 

Accreditability 

Legality 

Sustainability 

Balance 

Manageability  of  risk 

Timeliness 

Commitability 

Communicability 

Management  visibility 

Traceability  with 
requirements 

Sources:  DoD  acquisition  strategy  guidance  and  instruction  documents 


What  our  data  showed 


Acquisition  Quality 
Attribute 

Frequency 

Flexibility 

23 

Performability 

15 

Realism 

14 

Affordability 

10 

Survivability 

6 

Executability 

5 

Responsiveness 

4 

Programmatic 

Transparency 

2 

Innovativeness 

1 

Schedulability 

1 
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Acquisition  Quality  Attribute  Scenarios 


Expressing  AQA  scenarios  similarly  to  software  QA  scenarios  is  a 
viable  path 

Scenario  from  software  domain: 


Software  QA 
Scenarios 

Acquisition  QA 
Scenarios 

Software 

architecture 

Acquisition 

strategy 

System 

Program 

Architect 

Program  manager 

Stimulus:  An  internal  component  fails 


Environment:  During  normal  operation 


Response: 


The  system  is  able  to  recognize  a  failure 
of  an  internal  component  and  has 
strategies  to  compensate  for  the  fault 


Scenario  from  acquisition  domain: 

Stimulus:  An  unexpected  budget  cut 


Environment: 

Response: 


For  a  multi-segment  system 

The  program  is  able  to  move  work  between 
major  segments  to  speed  up  or  slow  down 
separate  segments  within  the  available 
funding 
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What  can  AQA  scenarios  tell  us? 


Fundamentally,  AQA  scenarios  can  be  used  to 

•  Express  business  and  mission  goals  in  a  way  that  directly  influences  the 
acquisition  strategy 

•  Determine  the  appropriateness  of  the  acquisition  strategy  with  respect  to  any 
given  scenario 

Specifically,  3-  and  6-  part  AQA  scenarios  can  be  used  to  identify 
possible  incompatibilities  between 

•  AQA  scenarios 

•  Software  QA  scenarios  and  AQA  scenarios 
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Incompatibilities  between  Scenarios 


Stakeholder  A:  advocates  use  of  open  Stakeholder  B:  is  responsible  for 

source  software  as  a  means  of  ensuring  that  the  deliverables  meet 

increasing  responsiveness  to  users  rigorous  safety  standards 


Stimulus 

Users  request  significant  new 
functionality  to  be  delivered 
rapidly 

Stimulus 

A  new  requirement  to  adhere 
to  a  rigorous  safety  standard 
is  applied  to  the  system 

Environment 

during  the  program's 
development  phase 

Environment 

during  the  program's 
development  phase 

Response 

create  the  functionality  rapidly 
by  reusing  open  source  and 
software  from  other  projects  to 
provide  much  of  the  capability. 

Response 

The  developers  remove  all 
unreachable  code  to  insure 
that  the  system  will  pass 
stringent  new  certification 
standards. 
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Prototype  Elicitation  Workshop 


Adapted  the  QAW  for  eliciting  software  quality  attributes 

•  Greater  emphasis  on  the  business  goals  and  objectives  presentation 

•  Replaced  the  architecture  presentation  with  an  acquisition  strategy  and  plans 
presentation 

Conducted  the  prototype  on  a  real  program  using  SEI  staff  that  were 
supporting  the  program 

•  Generated  20  acquisition  QA  scenarios 


Acquisition  Quality 
Attribute 

Scenario 

Potential 

Acquisition  Tactic 

Flexibility 

The  user’s  system  requirements  change  radically  30  days 
before  the  RFP  is  released  when  the  “go  live”  date  is  fixed; 
the  RFP  is  released  regardless. 

Establish  fallback  strategies 
that  protect  the  “go  live” 
date. 

Affordability 

We  discover  that  the  cost  of  operating  the  system  will  be 
higher  than  the  ceiling  mandates  during  development  but 
before  initial  fielding;  the  system  (including  its  architecture) 
is  shifted  to  a  less  costly  alternative. 

Emphasize  the  need  for 
architecture  adaptability  and 
flexibility. 
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Phase  3:  Develop  and  Pilot  an  Alignment  Method 


Research  questions  that  are  focusing  our  work  this  year 

•  Can  business  goals  that  represent  the  full  range  of  program  stakeholders  be 
explicitly  defined  and  prioritized? 

•  Will  having  a  more  complete,  explicit  set  of  business  goals  generate  a  more 
complete  set  of  AQA  scenarios? 

•  Will  reconciling  Acquisition  QA  scenarios  and  Software  QA  scenarios  lead  to 
mutually  constraining  acquisition  strategy  and  software  architecture? 

•  Will  a  method  that  aligns  Acquisition  QAs  and  Software  QAs  be  useful  to  a 
program? 


Software  Engineering  Institute  Carnegie  Mellon  University 


2014  SEI  Research  Review 

11-12  February  2014.  Brownsword,  Place, 

Albert,  Carney 

©2014  Carnegie  Mellon  University 


12 


Phase  3  Research:  Work  to  Date  in  FY14 


Business  Goal  Determination 


Focus  on  capturing  business 
goals 


>  Identify  stakeholders 

>  Elicit  business  goals 

>  Represent  goals  in  standard 
form* 

Analyze  goal  subjects  and 
objects  to  identify  additional 
stakeholders 

Expect  the  PM  to  carry  this  out 


*Business  Goal  Scenarios  found  in  SEI  TN  CMU/SEI-2010-TN-018:  Probably  applies  to  Mission  Goal 

“Relating  Business  Goals  to  Architecturally  Significant  elicitation 

Requirements  for  Software  Systems" 
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Phase  3  Research:  Work  to  Date  in  FY14 


Quality  Attribute  Consistency  Focus  on  consistency  of 

scenarios 


>  Just  beginning  this  work 
Hypotheses 

>  Needs  reasonably  complete 
scenarios 

>  Will  require  feedback  to 
stakeholders  if  goals  have  to  be 
modified 

>  Performed  by  PM  and  evaluation 
team 
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Conclusion 


We’re  making  progress 

There  is  more  work  that  could  be  added  to  this  year’s  effort 

•  An  assessment  instrument 

•  Metrics 
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